SD6337/S92307 
October 24, 2000 



FRACTIONAL SCREEN VIDEO 
ENHANCEMENT APPARATUS 



INVENTORS: 



Barry L. Spletzer 

p 9504 Arvilla, N.E. 

5 Albuquerque, New Mexico 87111 

r= George S. Davidson 

1517 La Tuna, S.E. 
It Albuquerque, New Mexico 87123 

L_5 



- Daniel J. Zimmerer 

^ 42 Campo Rd.. 

u Tijeras, New Mexico 87059 

fy 

c3 Lisa C. Marron 

C3 405 Mankin St., N.E. 

Albuquerque, New Mexico 87123-1124 



Express Mail No.: EL027381 824US 




SD6337/S92307 
SPLETZER 



Fractional Screen Video Enhancement Apparatus 

This invention was made with Government support under Contract DE-AC04- 
94AL85000 awarded by the U. S. Department of Energy. The Government has certain 
rights in the invention. 

Background of the Invention 

This invention relates to the field of displays, specifically displays using multiple projectors to 
provide enhanced resolution. 

Conventional displays can be characterized by their resolution. Generally, higher resolution 
means higher display system cost. In many display applications, however, higher resolution is 
needed for only portions of the overall image. For example, in display of a sporting event, only a 
limited region (e.g., the pitcher and batter in a baseball game) benefits from high resolution at any 
given time. As another example, in a display of a computer image, only the portion currently being 
inspected needs to be displayed in high resolution. In both examples, low resolution display of the 
surrounding context can aid understanding of the portion displayed at high resolution. Also, the 
portion that benefits from high resolution display can change with time (the batter hits the ball and 
begins running the bases in the baseball game example). A conventional display with sufficient 
resolution for the high resolution portion can be more expensive; a conventional display at lower 
cost can be incapable of the high resolution needed for some portions of the image. 

Accordingly, there is a need for a display method and apparatus that can display portions of an 
image at a higher resolution and the rest at lower resolution, without incurring the cost associated 
with displaying the entire image at the higher resolution. 
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Description of the Figures 

The accompanying drawings, which are incorporated into and form part of the specification, 
illustrate embodiments of the invention and, together with the description, serve to explain the 
principles of the invention. 
5 Figure 1 is a schematic representation of a video display system according to the present 
invention. 

Figure 2 is a schematic representation of a video display system according to the present 
invention. 

Figure 3 is an illustration of geometrical relationships relevant to the present invention. 
4:0 Figure 4 is a schematic representation of image projection associated with the present invention. 

*9 Detailed Description of the Invention 

^3 The present invention provides a method and apparatus for displaying two portions of an 

y3 image at two resolutions. For example, the invention can display an entire image at a first 

H> resolution, and a subset of the image at a second, higher resolution. Two inexpensive, low 

?==? 

n|5 resolution displays can be used to produce a large image with high resolution only where needed. 

Figure 1 is a schematic representation of a video display system 10 according to the present 
"* invention. Video display system 10 comprises first 1 1 and second 12 video sources, arranged 
relative to a display medium 14. First video source 1 1 accepts input corresponding to a first 
portion 16 of an image to be displayed. An image transformer 13 accepts input corresponding to a 
20 second portion 17 of the image to be displayed and information defining the configurations of the 
first 11 and second 12 video sources and the display medium 14. Image transformer 13 
generates a signal input to second video source 12 compensated for the configurations such that 
the resulting display of the second portion on the display medium is aligned with the display of the 
first portion on the display medium. 



Q 
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First and Second Portions of the Image 

The first and second portions of the image can be any portions of the image, including the 
entire image. For example, the first portion can be an entire image to be displayed, and the 
second portion can be a subset of the entire image that requires display in higher resolution. The 
5 first portion can be, for example, a display of a camera image while the second portion is a subset 
of the camera image deserving specific attention (e.g., where motion occurs, or where detailed 
features are present). The first and second portions can change with time. In the entire 
image/subset example, the entire image can change with time, as with a panning camera or an 
animated computer display. The subset can also change with time, both in content (as the image 

^0 itself changes) and in relation to the entire image (as the subset deserving attention moves 
relative to the image). Motion of the second portion relative to the entire image can follow, as 

^ examples, a viewer's eye movements, motion of a pointing device, and motion of an onscreen 

Li ! 

£0 object (such as a ball in a sporting event). 

= Display Medium 

M 

CJ5 The display medium can be of any form compatible with the video sources. As examples, rear 

ru 

j= projection displays, screens, and suitable wall surfaces can each serve as a display medium. 
First Video Source 

Those skilled in the art know a variety of suitable video sources suitable for use in various 
settings. Any combination of display technology that would be capable of displaying 2 different 

20 resolutions on the same screen may be used. Examples of suitable image display sources include 
rear projection systems, cathode ray tube (CRT), LCD, Digital Light Projectors (DLP), and plasma 
displays. The first video source can be operated in a conventional method to display a larger, 
lower resolution image. The mounting and alignment of the two video sources depends on the 
type of display sources, optical properties (such as focal length and depth of focus), image 

25 steering apparatus, and the geometry of the system. 
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In the example implementation described below, the two projectors were rigidly mounted to a 
base to reduce the degrees of freedom of the system. The main projector was set approximately 
perpendicular to the display screen to limit distortion of the main image. However, this is not 
necessary. The transform (T L ) compensates for linear distortion in overall transform. If the 
5 projection surface (screen) is planar, then homogenous transforms can be used. Otherwise, more 
complex transforms can be required. 

The second projector was mounted so that the focal range would coincide with the display 
screen after reflection from the mirror, and such that the optical axis of the lens intersects the 
mirror coordinate system at the origin. The angles between the projectors are arbitrary, and were 
1 0 chosen to restrict the overall size of the apparatus. 

m Second Video Source 

The second video source can be any of a variety of video sources as discussed above. The 

it second video source can be mounted so that its display corresponds to only a subset of the 

j ,=? region covered by the display of the first video source. If the second portion of the image is mobile 

jj> with respect to the first portion, then the second video source must be able to adjust the 

rjf placement of its display relative to the display of the first video source. 

Q A conventional video source can be combined with an image steerer to provide the second 
video source. The image steerer directs the output of the conventional video source to the 
appropriate portion of the display medium. As examples, an image steerer can comprise a 

20 moveable mirror, a controllable optical transmitter, or a micro-electromechanical device. A 
conventional video source can also be mounted mobile with respect to the display medium, 
allowing the output to be directed to the appropriate portion of the display medium by movement 
of the video source. 

Image Transformer 

25 The second portion of the image travels a different path to the display medium than does the 
first portion. Accordingly, the display of the second portion can be adjusted to achieve proper 
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registration between the two portions. In the discussion below, it is assumed that the second 
portion is directed toward the display medium by a gimbal-mounted mirror, controlled by computer 
commands to stepper motors, as shown in Figure 2. Those skilled in the art will appreciate similar 
image transformations suitable for other mechanisms directing the second portion to the display 
5 medium. 

The output of the second video source can undergo significant distortions depending on the 
location on the display medium to which it is projected. These distortions can include skew, 
rotation, and anisotropic stretching. The second portion can be transformed before reaching the 
second video source so that, after all the distortion, the display on the display medium is properly 
s 10 registered. The transformation can be complex, but, since ray tracing from the object plane of the 
« second video source to the display medium involves classical projection, rotation, and reflection, 

*3 the transformation can be expressed as a single homogeneous transform. Development of such 

i p| 

£3 transforms is discussed below. 

s " Basic homogeneous transforms 

Li 

H£ Homogeneous transforms can provide a powerful mathematical tool for converting between 

j= coordinate systems and determining the projection of images. A homogeneous transform 

comprises a 4x4 matrix that operates on a four element homogeneous coordinate vector. The first 
three elements of the homogeneous coordinate vector are the usual x, y, and z components used 
to describe a three-dimensional point. The fourth component (d) allows projection information to 

20 be carried along with the three-dimensional point. For a point in actual space this fourth 
component is always equal to 1 . The usefulness of this fourth component and its physical 
meaning will become more clear later. 

Homogeneous transforms are true matrices. Accordingly, they are amenable to various matrix 
operations; for example, several simple transforms can be multiplied together to produce a more 

25 complex transform. A homogeneous transform can be understood by examining some of the 
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basic components that can contribute to the transform. The first of these is translation of the 
coordinate system origin. An origin translation transform has the form shown in Equation 1. 

Equation 1 
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The transform of Equation 1 converts a general point in the s coordinate system to the b 
coordinate system. Examining the operation shown in Equation 1, the general point in the vector t 
is simply added to the point expressed in the s system. This constitutes a translation transform 
where the vector t is defined as the vector from the origin of the b system to the s system. Rather 
than writing on all the elements explicitly, this translational homogeneous transform can be 
defined as in Equation 2. 

ri 0 0 t ~\ 

Equation 2 

u [ o / 

In addition to translation, rotation between two coordinate systems can be defined by the 
homogeneous transform of Equation 3, where C is a point expressed in the c coordinate system 
and B is the identical point expressed in the b coordinate system. 
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B = 



b y -c x b y -c y b / -c 2 0 
4-^ b 2 c y b 2 c 2 0 



Equation 3 



0 0 0 1 

All of the elements of this rotation vector are made up of dot products between the unit vectors 
of the b and c coordinate systems. The notation used here for this rotational homogeneous 
transform is shown in Equation 4a. 

Equation 4a 
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The rotational transform and origin translation transform can be combined to perform a 
complete coordinate system transform as in Equation 4b. 

C(b,c) = 0(b,c)R(b,c) Equation 4b 

In addition to rotation and translation scaling may be accomplished by a homogeneous 
transform. Simple isotropic scaling is done by multiplying each of the three components by a 



fixed scale factor f. The scaling transform is defined as in Equation 5. 

"/OOO" 



S(/) = 



0/00 
0 0/0 
0 0 0 1 



Equation 5 



The next basic transform is a projection transform which uses the bottom row of the 
homogeneous transform. Before presenting the form of this transform, some explanation of the 
fourth element (d) of a homogeneous coordinate can be useful. 
[h As mentioned before, a point in space can be represented by the usual xyz coordinates and a 
TO d value of 1 . The definition of homogeneous coordinates states that any four element coordinate 
can be converted into a point in space by dividing all four elements by the value of d. By defining 
this as a valid operation on a homogeneous coordinate the ratios between the first three elements 
are invariant. This means that a homogeneous coordinate describes a line passing through the 
origin from which the point is measured and through the actual point in space. This property 
15 makes homogeneous coordinates very useful for calculating projections. If the origin of the 
. coordinate system is moved to the observation point or perspective point then the homogeneous 
coordinate of a point defines a line that passes through the point in space and through the 
observation point. By definition, this line is the projection of the actual point through the 
observation point. 

20 Using the projection transform, the homogeneous representation with respect to the origin can 
be produced for any point. By adjusting the fourth coordinate, any point along the line passing 
through the original point and the origin can be represented. Specifically, consider the value of the 
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fourth coordinate required to represent the point where the line intersects a plane. The plane is 
completely defined by the vector N that connects the origin to the image plane and is normal to 
the image plane. Figure 3 illustrates the geometry of this situation. The original point C is 
measured with respect to the origin placed at the observation point. The vector C p is the 
projection of the point onto the plane. 



Similar triangles lead to Equation 6. 

CN C 

" = C P 

The vector C can be scaled to represent C p as in Equation 7. 



C_ N N-N 

c =c— = c — - = c — = c 

p C CN CN 
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N 



Equation 6 



Equation 7 



By the definition of homogeneous coordinates, the fourth coordinate (d) can be divided into the 
homogeneous representation to create a point in space. Therefore, to convert the point C to the 
point Cp, only the value of d need be modified. The value of d required to perform the conversion 
is given by Equation 8. 

C-N 

rf = — = C M Equation 8 



N-N 

The vector M can be completely determined from the projection plane definition N as in 
Equation 9. 



N 
N-N 



Equation 9 
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The bottom row of the homogeneous transformation can be used to produce this value. The 
first three components of the bottom row operate on the vector to produce a dot product. The 
fourth component must be 0 so that the proper value of d is obtained. This results in a transform 



of Equation 10, which is the complete projection transform. 

1 0 0 0" 
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Equation 10 



n n n n n n 

The final basic transform is the zoom transform, which consists of the identity matrix with a 
zoom value in the lower right element. The zoom value is a multiplier of the input d value of the 
homogeneous coordinate. Since the d value can be divided into the homogeneous coordinate to 
produce a point in space the zoom factor is the isotropic scaling of the distance from the origin to 
the input point. After the transform, the distance to the point is 1/z times the distance to the 



original point. The zoom transform as defined as in Equation 11. 

'1 0 0 0l 



Z(z) = 



0 10 0 
0 0 10 
0 0 0 z 



Equation 1 1 



20 



One more transform is needed to completely analyze the system. This is the transform 
resulting from reflection in a mirror. The nature this transform will be determined here based on a 
vector analysis of reflection. The position of the mirror can be completely described by the normal 
vector (n) to the mirror. In general this normal vector will be known and the resulting transform is 
desired. Unlike previous transforms which expressed a given point in a second coordinate 
system, the mirror transform takes a point in the original coordinate system and expresses 
location of the reflected image of that point in the original system (virtual point) 

The development of a transform to handle mirror reflection requires the determination of the 
relationship between the incident coordinate system and the reflected coordinate system. To 
simplify this determination, place the origin of both coordinate systems coincident on the mirror. 
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The z-axis (k) of the incident system points towards the mirror. The required rotation to produce 
the reflected system must be determined. The notation for the reflected system will be to add a 
prime () to the notation of the incident system. The law of reflection gives Equation 12. 

k' = * - (2* • n)n = k- 2n z h Equation 1 2 

The equation of the mirror plane is given in Equation 13, where A is any vector lying in the 
mirror plane. 

A ■/! = () Equation 13 

To determine the direction of the reflected x-axis, consider the projection of a ray parallel to the 
z-axis but displaced one unit in the x direction. The intersection of this ray and of the z-axis with 
the mirror creates two points, which define the projection of the x-axis onto the mirror. The vector 
representing this projection can be determined by noting the equation of the incoming ray, given 
WCp in Equation 14. 

Li! 

CS x = \, y = 0 Equation 14 

Solving for the z coordinate of the intersection of the ray with the mirror plane yields 
Equation 15. 

= 0 Equation 15 



n. 



The projection of the x-axis onto the mirror (/J has the above x, y, and z components as in 
Equation 16. 

~ n x - 

L =i ~~ k Equation 16 

15 This vector, when viewed from the /c'direction, constitutes the x-axis. The i m vector can be 
projected into the plane perpendicular to the /c'direction by subtracting the component of the 
vector parallel to k'. This results in the expression for /'in Equation 17. 

I ' , = l ™ Equation 17 
Equation 17 shows /'as a unit vector. This is true because an object is not foreshortened by 
reflection So that the magnitude of the reflected vector is the same as the incoming vector as long 
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as the view direction relative to the vector remains the same. Combining this expression with the 
above relationships gives Equation 18. 
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Equation 18 
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Evaluating some useful subexpressions gives Equation 19. 

i .*'=/•[*- (2k ■ n)n] = -(2k . n)(i • it) Equation 1 9 

k'k'=k\k-(2k'n)fi\ = \'2(k-n) 2 

Combining yields Equation 20. 

/ ' = / - ^-k + 2(k ■ n)(i ■ n)k' + ^ 1 - 2(k> n\ ]*' Equation 20 

/ ' = f - ^2(k • h)n + 2(* ■ • n - (it ■ n)^j k' 

i ' = i 2(k • n)h + 2(k ■ nfi -n-(k- ")^)[* - (2 k • n)n\ 

Since all the indicated dot products with the mirror normal have one of the original unit vectors 
as a component, they can be replaced by the components of the mirror normal unit vector as in 
Equations 21 and 22. 

[£-(2w a )«] Equation 21 



i'=/--^2iyi + 2/i a 



n 



i ' = i ~2n x n 

J'=J- 2n y n Equation 22 

From this, the homogeneous transformation matrix of a mirror can be developed. This matrix 
consists solely of rotation with the origin of both systems located coincident on the mirror surface 
as shown in Equation 23. 

1-?w 2 —On n -9m « Hi 

Equation 23 
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The notation and description of each of the required contributing transforms is summarized in 
Table 1. 

Table 1 



TranQfnrm 
I I ai laiui in 


<jy 1 1 iuui 




Orinin translation 
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Coordinate rotation 


R(b.c) 


Converts coordinates from c system to b system provided 

LlldL Lilt; oybltJlll UliyiMo die UUIMulUGIIl 


Coordinate conversion 


C(b,c) 


Convert coordinates from c system to b system accounting 
for both rotation and translation 


isotropic scaling 




isoiropicaiiy scales cooruinaie values uy lacior r 


Projection 


P(N) 


Projects points from three-dimensional space onto an image 

olan<5> VA/horo M ic tho \/OPtor from tho r c\(~\r c\\T~\a\ a e\/c tarn r\rinir» 
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to the image plane and normal to the image line 


Zoom 


Z(z) 


Scales three-dimensional points so the resulting point lies 
along the original line connecting the origin with the point but 
at a fraction 1/z from the origin 


Mirror reflection 


M(n) 


Determines the image of the original point has seen in a 
mirror based on the unit normal n to the mirror 



Image projection using homogeneous transforms 

Before tackling the more difficult problem of two projectors and a mirror, consider the problem 
of producing a projection of an object onto an image plane. This requires that all rays from the 
object to the image plane pass through the pinhole or observation point. The intersection of such 
a ray with the image plane is the projected image of the original point. Figure 4 shows this 
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schematically. A single homogeneous transform can be constructed to perform this operation by 
combining several of the previously discussed basic transforms. 

In the figure, the points in the object are originally expressed in the S coordinate system. The 
homogeneous coordinate representation of the general vector S is given in Equation 24. 

Equation 24 



S = 



s y 
s, 

1 



The 1 for the fourth element means that the actual values of the vector S are used here. First, 

construct a simple rotation and translation transform to convert from the S coordinate system to 

the C coordinate system with the origin at the pinhole. This transform is in Equation 25. 

JfjJ C = C(c,s)S Equation 25 

In this transform, c and s refer to the appropriate unit vectors of the two coordinate systems 

and s 0 is the vector from the origin of c to the origin of s measured in the c coordinate system. 

Using this transform, the homogeneous representation with respect to the pinhole can be 

produced for any point in the original object. By adding a projection transform the coordinates of 

any point in the image plane can be determined, as in Equation 26. 

C p = P(N)C(c,s)S Equation 26 

S ^ This gives the coordinates of all points projected onto the image plane in the pinhole 

coordinate system. One final conversion is needed and that is to convert the coordinate system 

15 reference to the image plane (b). This transformation has the same form as the transformation 
between the s and c systems earlier. Adding the appropriate rotation and translation transforms 
results in Equation 27. 

B = C(MP(N)C(c, j)S Equation 27 

This result provides the homogeneous coordinates in the image plane coordinate system. In 
general the value of the fourth element (cO will not be 1.0 and all elements in the resulting vector 
20 must be divided by d to yield the actual point in space. 
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Experimental determination of homogeneous transforms 

The previous derivation shows that a single homogeneous transform does exist to perform the 
projection. However, since it is constructed from many basic transforms multiplied together, the 
relatively simple components that make up each transform and the usual association of areas of 
the transform with rotation, translation, perspective, and zoom will in general be lost. The overall 
transform can be determined experimentally without any advance knowledge of the contributing 
parameters. Equation 28 gives the completely general form of a homogeneous transform. 
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Point-by-point measurements in the s and b coordinate systems can be made to 
experimentally determine the transform for a given system. Each point measurement will yield 
three new scalar equations but introduce one additional unknown, that being d which is unique for 
each point. Because of this, each measured point will provide information to solve for two 
additional unknowns. The set of equations resulting from these measurements is homogeneous. 
However, each element in the transformation matrix can be divided by a constant without altering 
the result. This is because of the scaling factor of. Dividing each element by a constant divides the 
resulting value of d and the values of b x d, byd, and b 2 d by the same constant so that the 
determination of the final coordinate is unaffected. Doing this results in the set of linear equations 
for each measured three-dimensional point shown in Equation 29. 

Equation 29 
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The 16th value of the transformation matrix, t 44 , is set to 1.0. A set of these three independent 
equations can be constructed from any measured point where a coordinate in the S system is 
measured and corresponds to a known point in the B system. This way, in the general case, at 
least five points can be measured to completely determine the transformation. Since all the 
equations are linear with respect to the unknown coefficients, linear regression 
techniques (discussed later) can be used to reduce the error in the resulting transform by 
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measuring additional points. In the specific case of interest, points in both the S and the S 
coordinate systems are constrained to a plane so that b z and s z are always zero. This reduces the 
equations to be solved to Equation 30. 

Equation 30 
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This set of equations provides solutions for only eight of the required 15 components. The 
seven undetermined components are used only as multipliers of the z components or to 
determine the z value of component. This set of equations shows that a minimum of four points 
can be required to completely determine the transform between the two planar coordinate 
systems and, as before, additional points can be used to provide a best fit transform. 

The use of only a minimum number of points can allow small errors in measurement to 
introduce very large errors in the resulting transform. Measuring many extra points and 
performing linear regression on the data can reduce the errors. The determination of the 
homogeneous transform via linear regression can proceed as follows. For each point measured 
on the system the left and right-most matrices can be augmented by adding to new rows. For 
example, if only three points are measured than the resulting equations set is as in Equation 31 . 

Equation 31 
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Equation 32 represents this symbolically. 

s t= b Equation 32 

The solution for the transform elements t using linear regression is as in Equation 33. 

t = (s T s)" ! (s T b) Equation 33 

This solution for the transform works equally well for the two-dimensional case where the eight 
elements oft are solved or for the three-dimensional case that solves for all 15 elements. In either 
case, t 44 can be set equal to 1 without any loss of generality. 

Construction of the complete transform 

Combining the ability to experimentally determine the homogeneous transform for overall 
system with the property of combining basic transforms to produce a complex transform allows 
the needed transforms to be constructed. The method can be complex because the movement of 
the mirror alters the transform in a complex way. 

The determination of the complete transform by measuring individual points can be done for a 
transform between the two projectors just as well as from one projector to the screen. Several 
intermediate coordinate systems can be convenient for this analysis; these are shown in Table 2. 

Table 2 



Symbol 


System description 


Alignment 


b 


small image projector object plane 


z normal to object plane 


L 


large image projector object plane 


z normal to object plane 


s 


Projection screen 


z normal to screen, origin along the mirror z 
axis, s x parallel to m x and s 2 parallel to m z 


P b 


small projector pinhole 


Unimportant 


Pl 


large projector pinhole 


Unimportant 


m 


mirror system 


z parallel to screen normal, origin at mirror axis 




small projector virtual pinhole 


Unimportant 
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The reasons for the various coordinate system alignment will be explained in this section. The 
techniques for achieving this alignment will be explained later. 

For the final system a number of calibration points can be measured to determine a specific 
transform. This is not sufficient since the transform is a function of the mirror position. Instead, the 
5 transform resulting from the calibration points must be decomposed to allow subsequent 

transforms to be constructed as a function of mirror position without additional measurements 
being required. The complete transform at any position can be expressed as a product of three 
transforms. The first of these three is the right transform (T R ), in Equation 34, which transforms 
from the small projector object plane to the mirror coordinate system. 

T R = C(m 9 P b )C(P b9 b) Equation 34 

Jo This is a direct coordinate transformation where all coordinate systems are fixed relative to 
,n each other so the transform is constant. The second is the center transform (T c ), in Equation 35, 
f f ; which converts from the mirror coordinate system to the screen. 

£ 2 T c = C(5, W )0(w, J P v )P(N)0(/> v ,AM)M(n) Equation 35 

: This transform requires some explanation. Going through the steps in sequence from right to 

y left, the transform begins with a point on the small projector object plane expressed in the mirror 

ry 

=35 coordinate system. Applying the M(n) transform gives the coordinates of the image of the point as 
C3 seen in the mirror. The next transform, 0(P v ,m) translates the coordinate system back to the 
virtual pinhole. From here, a projection transform P{N) converts the original point to the 
coordinates of the point projected onto the screen. The vector N is defined as the normal vector to 
the screen from the origin of the virtual pinhole system which is the virtual pinhole itself. Part of 
20 the difficulty in determining this center transform is the fact that the position of the virtual pinhole 
changes with changing mirror position. After the points are projected, the origin translation 
transform 0(m,P v ) converts back to the mirror coordinate system. The final basic transform 
C(s,m) converts the projected points into screen coordinates. 
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The final transform is the left transform(T L ), in Equation 36, which converts from the screen to 
the large projector object plane. 

T L =C(L,P L )C(P L ,s) Equation 36 

This is similar in form to the right transform and is a coordinate transformation between two 
fixed systems. Here however the constant value of the transform is dependent on the screen 
5 position relative to the projector box. Unlike the right transform which need only be measured 
once for a projector configuration, this transform can need to be determined for each screen 
setup. 

This division into three transforms is chosen because the center transform contains all the 
variables resulting from the repositioning of the mirror and the remaining transforms are constant 
fSlO for a given screen and projector position. The total transform can be measured for any mirror 
*L! position, as represented by Equation 37a. 

:-; T measurcd = T L T c (n)T R Equation 37a 

y For any given value of the mirror normal (n) the measured transform can be determined. It is 

f desired to determine the constant value of both the left on right transforms and to discover a way 

U to construct the center transform from the value of the mirror normal vector. Rearranging 

i y 

=p5 Equation 37a yields Equation 37b. 

□ TmcasuredTR -1 = T L T c( n ) Equation 37b 

For a given screen setup the left transform can be determined empirically as described 
previously. Also, for a given known mirror position, the measured transform can be determined. 
Once this is done, the above matrix equation becomes 16 equations with 32 unknowns since 
each element on the left side of the equation is a linear function of the unknown elements in the 
20 inverse of the right transform and each element on the right side is a linear function of the 
unknown elements of the center transform. 

By moving the mirror and re-determining the transform no additional unknowns are introduced 
on the left side of the equation but a completely unknown center transform results. Therefore, 
each additional mirror position measurement adds 16 equations and 16 unknowns. Because of 



SD6337; SPLETZER; Page 19 



m 



o 

i:3 



15 



this the center transform must be examined more carefully to reduce the total number of 
unknowns. The right-most contributing transform to the center transform of Equation 35 is a mirror 
transform, which is known by the position of the mirror measured in the mirror coordinate system. 
The next transform is an origin translation which consists generally of three unknowns. However, 
the magnitude of the translation is the distance between the mirror and virtual pinhole coordinate 
systems which is equal to the distance between the mirror and the actual pinhole. Therefore, the 
magnitude of the translation is constant for all mirror and screen positions. Calling the pinhole to 



mirror origin vector p the transform becomes as in Equation 38. 

'i o o P ; 



0(P v ,m) = 



0 10/7, 
0 0 1 p z 
0 0 0 1 



Equation 38 



This means that this contributing transform only has two additional unknowns for each mirror 
position. The third transform is the projection. By the alignment of the mirror coordinate system, 
the screen normal vector N is parallel to the mirror coordinate z-axis so the transform can be 
represented by a single unknown, as in Equation 39. 

Equation 39 

u i u u 
p ( N )= 0 0 1 

.° ° i 

The next transform moves the origin back to the mirror coordinate system. Since the 
operations following the previous translation do not affect the orientation or scaling of the 
coordinate system, this translation is the negative of the previous one, and is shown in 
Equation 40. 

H 0 0 -nl 

Equation 40 

l) i u - 71 

0(m,/> v ) = 



1 


0 


0 


-p* 


0 


1 


0 


"Py 


0 


0 


1 




0 


0 


0 


1 
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The final transform converts between the mirror and the screen coordinate system. The 
stipulation that the mirror and screen coordinate systems are aligned considerably simplifies this 



transformation. The result is Equation 41. 

-10 0 



C(s,m) = 



0 



0 10 0 
0 0 -1 z a 
0 0 0 1 



Equation 41 



In Equation 41, z 0 it is the distance between the two coordinate systems. Multiplied together 



these contributing transforms result in Equation 42. 



T c (n) = 



- 1 ° f 

» ■ 

0 0 ^TT^-l 

N 

0 0 £ 



PxPz 

N 



'-Px 
PyPz 



Py- N 

Pz +Z 



N 



N 



Equation 42 



M(n) 



Noticing that N - p z +z Q yields Equation 43. 

Px Px Z o 



T c (n) = 



-1 0 

0 1 - 

0 0 

0 0 



Pz +Z o 

Py 

Pz +Z o 

0 

1 



Pz+*o 
Py Z o 

P Z +Z o 

0 

Pz 



Equation 43 



M(n) 



Pz +Z o Pz +Z o _ 

The values of p x , p y , and p z change with mirror position. However, the values can be 
transformed from the virtual pinhole to the real pinhole by Equation 44. 

P = M(n)p Equation 44 
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In Equation 44, P is the fixed vector from the origin of the mirror system to the actual pinhole 
and p is the vector to the virtual pinhole. Performing the transform and explicitly substituting the 



values into the center transform yields Equation 45. 

P x -2n x Vn (P x -2n x V-n)z 0 



T c (n) = 



-1 0 



0 1 - 



P z -2n 2 Vn+z 0 
P y -2n y Pn 



0 0 
0 0 



P 2 -2«,P-n + z 0 
6 

1 

P - In P-n-fz. 



P z - 2H 2 P-n + z 0 
(P y -2n y ?>n)z o 

P z ~2n,V n + z G 

"o 

P, - 2« P • n 



Equation 45 



M(n) 



P z - 2« z P-n + z 0 

At first glance this seems to have complicated the transform. This is not the case. The previous 
transform had three unknowns which vary with mirror position. The unknowns in this transform 
are all invariant with mirror position. This means that the determination of the total transform at 
multiple mirror positions introduces no additional unknowns and therefore the unknown fixed 
values comprising the three contributing transforms can be determined with sufficient 
measurement. Because the relationships between the unknowns do not result in linear equations, 
the determination of P and z G is still not a trivial task. 

The values for P and z 0 can be determined by using a two-step linear least squares approach. 



First define the unknown matrix above as in Equation 46. 
P x -2n x ¥ n {P x - 2n x V ■ n)z 0 



-1 0 



0 I -■ 

0 0 
0 0 



P z -2/i.P n + z G 
* P - 2/i P n 



P r -2n z P-n + z 0 
6 
1 



P. -2/i,P-n + z„ 



P z -2/z.P n + z 0 
fc-2*>-n)z 0 . 

P 2 -2n z Yn + z 0 

P z -2M.P n 
P. -2/i.P-n + z rt 



-l o 4 

0 \ -A 2 

0 0 0 

0 0 A, 



0 



Equation 46 



= A 



Now the transformation takes on the form of Equation 47. 



L 1 measured A R — 



-1 0 A, 
0 1 -A 



0 0 
0 0 



0 
A, 



~ A l Z 0 

A 2 z 0 
0 



Equation 47 



M(n) 
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As discussed before T L it can be measured using the large format projector and any suitable 
screen. For any given mirror position (n) T measured is determined by correlating positions on the two 
projector images as measured at the screen. This means that the remaining unknowns are the 
elements of T R "\ A u A 2 , A 3i A 4 , and z 0 for a total of 21 unknowns. The transform T R " 1 is invariant * 
with mirror position but A is depending on the mirror position. This means that a sufficient number 
of equations can be generated by determining T measured at a suitable number of mirror positions. 
Each mirror position produces 16 equations. However, since each position only adds four 
unknowns (A u A 2l A 3l A 4 ) to the system, measurement of as little as two mirror positions can be 
sufficient to perform a least squares analysis and determine all the needed values. 

The use of multiple mirror positions gives multiple values for A, but the linear regression will 

determine the A matrix for each mirror position. Once the family of A matrices are determined, a 

second linear least squares can be used to determine the values of P and z 0 . This can be done 

because the determined elements of A can be combined with the algebraic definition of the matrix 

to construct a series of linear equations. These linear equations are in Equation 48. 

l — -2n x n y —2n x n s 0 
-2n x n y 1-2/2* -2n y n z 0 
-2n z n x -2n 2 n y \-2n] 1 

These equations can be used similarly to the technique described in empirical determination of 
a homogeneous transform to construct a linear least squares fit to the unknown values. 

Standard data analysis techniques may be used to calculate the regressions and transforms. 
Computer programming languages (C, basic, etc) or engineering calculation software (Mathcad, 
Matlab, etc) can be useful tools in this calculation. The display hardware determines the 
parameters for possible graphics software packages used in a particular system. For our 
example implementation LCD projectors were used, so any input compatible with the projector 
was sufficient. If the 2 nd projector were mounted to a pan and tilt platform, similar transfomations 
could be used. The transforms could be derived using a platform coordinate system instead of 
the mirror coordinate system. A similar system of matrices would result, with the absense of the 



z 



J_ 



Equation 48 
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mirror reflection transformation and the inclusion of the motion platform coordinate conversion 
transformation. 

Other configurations could be similarly derived. Consideration should be given to choice of 
coordinate systems, alignment of coordinate systems, degrees of freedom of the system, choice 
of steering system, and transformations of the optical paths of the images. If these system 
parameters for a particular setup are chosen carefully, the homogenous transforms can remain 
relatively straightforward. Additionally, the choice of lenses used with projection systems can be 
important. It is desirable to minimize radial or other non-linear distortion in both images. 
Nonlinear transformation schemes exist to compensate for some these distortions, but they can 
10 be difficult to implement. 



Example Implementation 



Ln A system according to the present invention is depicted in Figure 2. The system used two 
Proxima (trademark of Proxima Corporation) 9200 projectors, each having a maximum image 
resolution of 1024x728 picture elements. See "Proxima Desktop Projector 9200 User's Guide", 

Li 

^£5 Proxima Corporation. One projector was fitted with a short focal length lens; the other with a 
^ variable focal length zoom lens. The longer zoom lens was supported to prevent movement of its 
optical axis. 

A Model PTU-46-17.5 pan and tilt unit from Directed Perception, Inc., modified to provide 
access to the line where the pan and tilt axes intersect, was used to direct the high output of the 
20 second projector. See "Pan-tilt Unit User's Manual", Directed Perception, Inc. That model has a 
resolution of 0.0514° and can achieve speeds of 300° per second. The pan-tilt unit was controlled 
by a computer through a standard serial port. 

The second projector's optical axis was directed towards the center of rotation of the mirror. An 
aperture was used to block some of the high intensity light and provide a longer depth of focus for 
25 the long focal length lens. A V* diameter aperture was placed between the 1 st and 2 nd lenses 
inside the zoom lens assembly. This provided an adequate depth of focus. However, minor 
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interference patterns and color distortion can be present in this setup. Neutral density filters can 
be used to further reduce the intensity of the high resolution image. Two video inputs are 
necessary for the system, one for the main projector and one for the enhanced image. For an 
initial prototype and testing, two VGA laptop computers were used to produce the images and 
5 control the mirror assembly. The tested resolution was set at 640x480 pixels (due to the laptop 
capabilities) on each computer and projector. At this resolution, the projectors could be set to the 
expanded display setting. 

Calibration 

Those skilled in the art will appreciate various methods for determining the values suitable for 

fflD the transform. One example, used with the example implementation above, is described below. 

Cn Drafting vellum with 0.1 inch grid can be applied to a hard surface screen as a background for 

in the images and a means of recording data. The screen can be positioned so that it is 

p approximately parallel with the front edge of the system, and then aligned parallel to the mirror 

l~ surface. During the calibration of the system, the display screen can be aligned with the front 
u 

^5 surface mirror using a visible laser calibration tool. A laser can be mounted in a block such that 
\ : S the light emitted is perpendicular to the bottom surface of the block. The system can then be 

adjusted by placing the bottom of the block on the flat screen, and adjusting the screen (or 

projector apparatus or mirror) so that the display screen is parallel to a mirror position (when the 

laser reflects back upon itself). The position of the mirror at this alignment can be recorded, and 
20 then used to create the fixed mirror coordinate system. The pan-tilt can then be rotated 90° about 

the pan axis, and the point where the tilt axis intersects the screen located. This defines the 0,0 

screen coordinate for measurement of the data. 

A grid of data points (spaced at 40 by 40 pixels over the entire image) can be projected to the 

screen through the mirror, and to the screen from the main projector. Each grid point can be 
25 recorded on the vellum, and the mirror position noted. Once a few mirror position grids and the 

main grid are recorded, the physical x.y coordinates of each grid point can be written into a data 
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file along with the pixel location of each point. The amount of grid points is much greater than 
actually needed, but provides an over determined system for the linear regression technique for 
finding the measured transform from the measured data. The transform can then be found using 
the method described in the above. 
5 If P and Zq can be measured or estimated, then the transform can be determined without a 
double regression. The long focal length of a zoom lens on the second display source allows for a 
"pinhole" projection model to be used. An approximation for the focal distance of the lens can be 
calculated using Equation 49. 

W 2 fl 

*S In Equation 49, is the width of the LCD panel inside the projector, w 2 is the width of the 

JfjO image on the screen, ^ is the distance from the pinhole to the LCD, f 2 is the distance from the 

pinhole to the screen, Zo is the distance from the mirror center of rotation to the screen and z mb is 
the distance from the mirror center of rotation to the LCD panel (the magnitude of the constant P 
vector). Solving this system for f, can provide a reasonable measure of effective focal length of 
the lens. 

p5 Next, the "pinhole" vector P and the orientation of the enhanced image LCD can be found 

utilizing one position of the mirror and ray tracing. If it is assumed mathematically that the z axis 
of the screen and the LCD intersect at the mirror center of rotation, the screen coordinate of the 
x.y pixel located on the optical axis can be found in mirror coordinates. Reflecting this normalized 
vector from the mirror provides the unit direction of the optical axis. Since the magnitude is 
20 known, the P vector is now known. The screen coordinate of the pixel representing the maximum 
x value on the LCD can be used to determine the orientation of the x axis in mirror coordinates. 
Thus, the orientation of the LCD can be determined. 

From this, the T R matrix can be created using the equation T R =ABC where C consists of a 
seating and translation matrix to transform computer coordinates into inches, B is the rotation 
25 from the LCD system to the mirror coordinate system and A is translation from the LCD origin to 



La 
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the mirror coordinate system origin. This is the complete transform from the LCD to mirror 
coordinates, and is dependent only on the pinhole. This transform remains constant for a given 
screen setup. 

The next part of the transform is created using Equation 50. 

T c = C(s, m)0(m, Pv)p(n)0(Pv ) M)M(n) Equation 50 

Components of Equation 50 are given by Equation 51 , where p v is the location of the virtual 
pinhole in mirror coordinates, Zq is the distance to the screen, and n is the mirror normal vector in 



mirror coordinates. A new transform can be created for each mirror position.. 

-ZI_ __ 
x y 

\2 



M(n)= 



-2 (n V -2n n -2n n 0 

\ x/ x y x z 



-2n -n 1-2-fn f -2-n n 0 
x y \ yj y z 

-2-n n -2n n 1-2-fn ) 2 0 

x z y z \ z) 

0 0 0 1 

1 0 o - Pv 



Equation 51 



0(P v ,m)= 



P(N)= 



I 0 

0 1 

0 0 

0 0 



o i o - Pv 
o o 1 - Pv 

0 0 0 1 
0 0 



0(fn,P v )= 



-Pv + z 0 

z 

1 0 0 p v 
0 1 o Pv 
0 0 1 Pv 



C(s,m)= 



0 0 0 1 
-10 0 0 

0 10 0 

0 0-1 z f 



0 0 0 1 

Multiplication of this transform by the T L matrix (measured transform from screen coordinates 
to main screen computer coordinates) completes the transformation from enhanced image 
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computer coordinates to main screen computer coordinates. Inverting this completed transform 
provides the transform from main screen computer coordinates to enhanced image coordinates. 

In order to calculate where the enhanced image should be drawn, the mirror normal can be 
found based on the screen coordinate that the axis pixel should be located. Knowing this point 
and the location of the pinhole in mirror coordinates, the mirror normal can be found by 
subtracting the normalized vector to the pinhole from the negative of the unit vector to the screen 
position (all in mirror coordinates) and then normalizing the result. The pan and tilt steps to 
produce this normal can be found by solving the following system of equations. 

The relative angles to the mirror coordinate system can be calculated using Equation 52. 



In Equation 52, 9p = relative pan angle in mirror coordinates and 6t = relative tilt angle in 
degrees in mirror coordinates. The pan and tilt encoder steps can be found by Equation 53, 
where pan 0 and tiltg are the encoder steps used to define the mirror coordinate system.. 



The particular sizes and equipment discussed above are cited merely to illustrate particular 
embodiments of the invention. It is contemplated that the use of the invention may involve 
components having different sizes and characteristics. It is intended that the scope of the 
invention be defined by the claims appended hereto. 



-n x =sin(8p)cos(8t) 



Equation 52 



-n y =sin(8t) 
-n z =(cos(9p)cos(8t)) 



pan=- (Gp- 1 1 14.0849- pan Q ) 
tilt=9t l 114.0849- til^ 



Equation 53 



SD6337; SPLETZER; Page 28 



